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REMARKS 

Favorable reconsideration and allowance of the present patent application are 
respectfully requested in view of the foregoing amendments and the following remarks. 
Claims 1-29 are pending in the application. 

Specification Informalities 

The specification was objected to because of informalities. This Amendment 
addresses this objection by correcting the noted informalities in the substitute specification, 
in which no new matter has been added. Accordingly, withdrawal of this objection is 
respectfully requested. 

Additionally, Applicants respectfully traverse the allegation in the Office Action that 

one of ordinary skill in the art would not understand the invention from the application as 

filed. For example, the Office Action alleges that relation is unclear and that relationship is 

assumed. However, Applicants contend that the relation concept would be understandable 

to one of ordinary skill in the art. Particularly, in view of the examples in the specification 

itself. For example, the specification at page 8, line 25 to page 9, line 2 recites the following: 

Here, a constant relation means either a relation between characters 
that cannot change throughout a video, such as a parent to child relation, or a 
relation which is most representative of the relations between characters. 

Referring to the section referenced above, it is clear that a "relation between characters" 

represents relations such as the illustrated parent/ child relation. 
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Further, the term "relation" is used in Moving Picture Experts Group-7 (MPEG-7) 
and therefore one of ordinary skill in the art would most assuredly understand relation and its 
application to videos and video processing generally, and to video browsing systems 
specifically. 

Accordingly, although the specification has been amended to be more grammatically 
correct, Applicants respectfully submit that the invention was clearly described in the 
originally filed application to one of ordinary skill in the art. 

Claim Objections and 35 U.S.C. § 112 Rejections 

Claims 1-20 were rejected under 35 U.S.C. § 112, second paragraph, as being 
indefinite for failing to particularly point out and distincdy claim the subject matter set forth 
therein. Some of the formalities identified in the Office Action have been addressed in this 
Amendment, and Applicants accordingly request the Examiner to reconsider and withdraw 
this rejection in view of the foregoing amendments and following arguments. 

Regarding the term "relation", for reasons noted in the foregoing section, Applicants 
respectfully submit that the term "relation" does not render the claims indefinite, as alleged 
in the Office Action. 

Regarding the rejection to claim 2 in the Office Action, Applicants 1 respectfully 
traverse this rejection. The term summary data is clear on its face and Applicants do not 
believe that summary list is correct or provides any additional clarity. 
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Regarding claim 12, the term "DS" is defined in the specification as "description 
scheme" and Applicants have amended the claim to reflect this. Further, the Office Action 
alleges that "wherein the highlight view DS is organized into multiple levels which enables a 
display of multi-levels of highlight data and wherein the key frame view DS is organized into 
multiple levels which enables a display of multi-levels of summarized data" cannot be 
understood by one skilled in the art. Applicants respectfully disagree. For example, "multi- 
levels" is not incomprehensible and is merely an alternate term for multiple levels. Likewise, 
the key frame view is related to the highlight view as illustrated in Fig. 1, refs. 105 and 106 
and is described by concrete example on page 14, lines 1-24. 

The foregoing claim amendments to address the 35 U.S.C. § 112, second paragraph 
rejections and claim objections were made to correct formalities such as grammatical errors 
and to place the present Application in better form for examination. Therefore, the 
foregoing amendments do not narrow the scope of the pending claims. 

35 U.S.C. § 102 & 103 Rejections 

Claims 1-5 and 12-14 were rejected under 35 U.S.C. § 102(a) as allegedly being 
anticipated by Schein et al. (U.S. Patent No. 6,323,911, "Schein"). Claims 6-11 and 15-20 
were rejected under 35 U.S.C. § 103 (a) as allegedly being unpatentable over Schein et al. 
(U.S. Patent No. 6,323,911, "Schein") in view of Yeo et al. (U.S. Patent No. 5,831,945, 
"Yeo"). Applicants respectfully traverse each of these rejections for at least the following 
reasons. 
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Regarding the rejection to claim 1, the Office Action has relied upon Schein Fig. 4A, 

ref. 102 to show a video browsing system. However, as disclosed in Schein, reference 102 is 

a channel program schedule (i.e., a guide for selecting and getting information on programs 

on different channels), which is disclosed in column 9, lines 1-9 as follows. 

As shown in FIG. 4A, program guide 102 preferably includes a schedule 
information area 106 having a program matrix 108 of cells or items that depict 
the shows that are being presented on each channel at each time during the 
day. Program guide 102 conveniently lists the channels in a vertical column to 
the left of the program matrix 108 and the times in a horizontal row above 
matrix 108. As shown, the viewer may vertically scroll through a particular 
time or horizontally scroll through a channel. 

As clearly described in the foregoing section, reference 102 of Schein is a program guide and 

not a video browsing system (e.g., a system to browse the content of a video). Therefore, at 

least this element is not shown in Schein as alleged by the Office Action. 

Likewise, the Office Action describes Fig. 4A, ref 126 as "a character screen which 

displays characters of the video" (Office Action, page 8). However, this* characterization is 

contradicted by Schein. In contrast to the characterization in the Office Action, Schein 

describes reference 126 in column 9, lines 38-45 as follows. 

A program area 126 depicts the currently tuned program and a preview 
window area 128 can be used for all types of promotional, descriptional, or 
contextual video or graphics, such as a short preview of the show that is 
currently being highlighted in show matrix 106. Preview window area 128 may 
also be interactional similar to the other areas of guide 102. 

As stated in the above referenced section, Schein discloses reference 126 as being a 

program area that "depicts the currendy tuned program". Once again, Applicants 

respectfully submit that the Office Action disregards the clear teachings of Schein and 
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has instead substituted a function that is not taught or suggested in the Schein 
reference. It is clear that Schein teaches area 126 is a video window that displays the 
currently tuned program, much like a picture-in-picture display. There is no 
suggestion or teaching that reference 126 is used as a "a character screen" as alleged. 

Still further, the Office Action appears to allege that reference 126 of Schein is 
also the main screen, although the Applicants are unsure of the correspondence 
alleged in regard to reference 126. However, the Office Action cannot rely on 
reference 126 to teach both a character screen and a main screen without some 
suggestion or teaching in Schein of which there is none. Additionally, the Office 
Action has not provided any teaching to address the specific features of claim 1 such 
as "video segments corresponding to significant events showing a relationship 
between two displayed characters of said character screen according to a user 
selection, wherein said relationship may be constant or variable." The Office Action 
and Schein are completely silent on these features. 

Regarding claim 12, the Office Action again relies on the Schein channel 
program guide, reference 102, to teach a video browsing system, which it clearly is not 
as discussed in the foregoing arguments. Further, the Office Action apparently 
alleges that reference 110 of Schein discloses the remaining features of Applicants 1 
claimed combinations (Office Action pages 9-10). Once again, the Office Action has 
not shown how one reference can account for all the claimed features, or provided 
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any support for this position. In contrast to the Office Action's interpretation, Schein 
discloses that reference 110 is merely "a cursor", (Schein, column 9, line 10). 

As stated in MPEP § 2131, "[a] claim is anticipated only if each and every 
element as set forth in the claim is found, either expressly or inherendy described, in a 
single prior art reference." Verdegaal Bros. v. Union Oil Co. of California, 2 USPQ2d 
1051, 1053 (Fed. Cir. 1987). "The identical invention must be shown in as complete 
detail as is contained in the ...claim." Richardson v. Suzuki Motor Co., 868 F.2d 1226, 
1236, 9 USPQ2d 1913, 1920 (Fed. Cir. 1989). The Schein reference neither expressly 
nor inherendy describes every feature of Applicants 1 claimed combinations as detailed 
in the foregoing arguments. Therefore, Applicants respectfully submit that the Shein 
reference does not anticipate Applicants' claimed combinations as alleged by the 
Office Action. 

Regarding claim 15, the Office Action again relies on the Schein channel 
program guide, reference 102, to teach a video browsing system, which it clearly is not 
as discussed in the foregoing arguments. Further, the Office Action relies on the 
faulty characterization of reference 126 already addressed in regard to claim 1. 
Therefore, the foregoing arguments in reference to claim 1 regarding reference 126 
are explicitly relied upon, without repetition herein. 

However, the Office Action admits that Schein fails to teach a tree structure 

and relies upon Yeo to cure this deficiency. Specifically, the Office Action alleges that 

Yeo teaches "a relation screen which displays constant relations and variable relations 
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between a character selected from the character screen and related characters, wherein 
said constant relations are displayed in a tree-like structure (fig. 4)." The Office 
Action then alleges that it would have been obvious to combine these two teachings 
to arrive at Applicants' claimed combinations. However, Applicants note that the 
Yeo reference fails to cure the additional deficiencies of the Schein reference, as 
discussed in relation to claim 1. 

As stated in MPEP § 2143.01, to establish prima facie obviousness of a claimed 
invention, all the claim limitations must be taught or suggested by the prior art. In re 
Royka, 490 F.2d 981, 180 USPQ 580 (CCPA 1974). "All words in a claim must be 
considered in judging the patentability of that claim against the prior art." In re Wilson, 
424 F.2d 1382, 165 USPQ 494, 496 (CCPA 1970). Neither Schein, Yeo nor the 
combination of these references discloses features of Applicants' claimed 
combinations as noted above. Therefore, these references do not render Applicants' 
claimed combinations obvious as alleged. 

Further, the combination the Schein and Yeo references are not properly 

combinable, as proposed by the Office Action. The Schein reference is directed to a 

channel program guide. In contrast, Fig. 4 of Yeo discloses an initial scene transition 

graph. Applicants respectfully submit that graphing an intitial scene transition 

provides no benefit to a channel program guide and one of ordinary skill in the art 

would not have been motivated to combine Fig. 4 of Yeo with Fig. 4A of Schein, as 

alleged by the Office Action. Further, adding a scene transition diagram to a channel 
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program guide, if possible, would at the very least change the principle of operation 
of the Schein reference. 

As stated in MPEP § 2143.01, if the proposed modification would render the 
prior art invention being modified unsatisfactory for its intended purpose, then there 
is no suggestion or motivation to make the proposed modification. In re Gordon, 733 
F.2d 900, 221 USPQ 1125 (Fed. Cir. 1984). Further, if the proposed modification or 
combination of the prior art would change the principle of operation of the prior art 
invention being modified, then the teachings of the references are not sufficient to 
render the claims prima facie obvious. In re Rati, 270 F.2d 810, 123 USPQ 349 
(CCPA 1959). Since the combination proposed by the Office Action would at least 
change the principle of operation of the Schein reference as discussed above, the 
references are not sufficient to render the claims prima facie obvious. 

Accordingly, Applicants respectfully request reconsideration and withdrawal of 
the rejections of independent claims 1, 12 and 15. New independent claims 21 recites 
related subject matter to the above-identified independent claims, and is therefore 
allowable for reasons similar to those given above. 

The dependent claims are allowable at least by virtue of their dependency on 
the above-identified independent claims. See MPEP § 2143.01. Moreover, these 
claims recite additional subject matter, which is not suggested by the documents taken 
either alone or in combination. 
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Further, Applicants submit that one of ordinary skill in the art would not have 
been motivated to modify the systems of Schein and Yeo to arrive at Applicants' 
claimed combinations absent impermissible hindsight reference to Applicants 1 
specification. Therefore, Applicants respectfully request reconsideration and 
withdrawal of the outstanding rejections of claims 1-20 and allowance of claims 1-31. 

CONCLUSION 

In view of the foregoing amendments and remarks, it is respectfully submitted that 
this application is in condition for allowance. Favorable consideration and prompt allowance 
are earnestly solicited. 

Should there be any outstanding matters that need to be resolved in the present 
application, the Examiner is respectfully requested to contact Mark E. Olds, to conduct an 
interview in an effort to expedite prosecution in connection with the present application. 
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To the extent necessary, a petition for an extension of time under 37 C.F.R. 1.136 is 

hereby made. Please charge any shortage in fees due in connection with the filing of this, 

concurrent and future replies, including extension of time fees, to Deposit Account 16-0607 

and please credit any excess fees to such deposit account. 

Respectfully submitted, 
FLESHNER & KIM, LLP 

Carl R. Wesolowski 
Registration No. 40,372 
Mark E. Olds 
Registration No. 46,507 

P.O. Box 221200 
Chantilly,VA 20153-1200 

(703) 502-9440 CRW/MEO:tljw:par 

Date: July 10, 2003 
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VIDEO BROWSER BASED ON CHARACTER RELATION ft£Q£\ygQ 

JUL 1 4 2003 

background OF the invention Technology Center 2100 

Field of the Invention 

The present invention relates to a video browser, and more particularly to a video 
browser for browsing a video based on relations between characters. 
Background of the Related Art 

Typically, users simply view movies and/ or dramas as broadcasted through a TV or 
played at a movie theate ftheatre . However, a user may wish to view a particular movie or 
drama at a particular time, or wish to view only a particular section of a movie or a drama. 
Accordingly, various techniques which enables a selective watching of a movie/drama or 
sections of a movie/drama have been suggested. 

In the related art, for example, various video data may be represented or classified 
into format ehunkgortion, index ehunkportion, media ehm kportion , segment ehtrn kportion , 
target ehtmkportion, and/ or representation eht±n kportion . Also, data on various characters or 
objects such as a name of an object, position on the screen, numeric data with relation to a 
segment of the video data in which the object appears, may be represented by the target and 
representation dmn kportion . Accordingly, a user can select an object through a table and 
reproduce for display a particular segment where the object is shown in the video. 

In another other related art, various additional data of a video data are obtained 
before, during or after the production of the video data. Thereafter, an additional 
information table of the obtained data is composed and provided to users. Namely, the 
additional data table may include a position where an actor appears, a position where a 



character of the actor appears, and a position where stage properties appear, such that a scene 
can be reproduced as selected by a user through the additional data table. For example, if a 
user selects a stage property, information on the selected stage property such as the 
manufacturer and price may be displayed on a screen, and the user may be able connect with 
the manufacturer or a seller of the stage property through a network connection. 

In still another other related art, recording information on each segment of a video in 
a video map has been suggested. That is, information such as the degree of violence, the 
degree of adult contents, the degree of importance of contents, characters positions, and the 
degree of difficulty in understanding may be indicated for each segment of a video in the 
video map. Thus, the user may set a degree of preference for one or more items of the video 
map, and only segments of the video meeting the set degree of preference would be 
reproduced, thereby limiting a display of particular contents to unauthorized viewers. 

Similarly, other techniques in the related art as described above provide items simply 
arranged without any relation to the objects appearing in the movie or drama, based upon the 
selection of the user. However, the contents of a movie or drama generally builds build 
around relations between characters, places and events. For example, relations between 
characters may not change from beginning to the end of the story or may continuously vary. 
Moreover, since one or more characters can relate to a specific character in the movie or 
drama, the browsing method in the related art substantially fails to provide an accurate 
understanding of the story of the movie or drama to the user. 

Therefore, techniques in the related arts have disadvantages in that it is difficult to 
understand a video centering on relations among characters according to the development of 
events, changes of relations, and relations among characters and places as events develop. 
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SUMMARY OF THE INVENTION 

Accordingly, an object of the present invention is to solve at least the problems and 
disadvantages of the related art. 

Another object of the present invention is to provide a video browser in which event 
segments showing changes in relations between characters are summarized and displayed 
based on a video data structure which includes characters in a video that significandy act on 
development of an event, and includes a constant and variable relations between characters. 

A still another object of the present invention is to provide a video browser in which 
main events that significandy act on variable relations between characters are connected with 
corresponding variable relations based on a video data structure which includes characters in 
a video that significandy act on development of an event, and includes a constant and 
variable relations between characters. 

A further object of the present invention is to provide a video browser in which a key 
frame of event segments which show a corresponding relation between characters is 
displayed based on a video data structure which includes characters in a video that 
significandy act on development of an event, and includes a constant and variable relations 
between characters. 

A further object of the present invention is to provide a video browser in which a 
constant relation and changes in relations between characters are displayed depending on 
preset relation types based on a video data structure which includes characters in a video that 
significandy act on development of an event, and includes a constant and variable relations 
between characters. 
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Additional advantages, objects, and features of the invention will be set forth in part 
in the description which follows and in part will become apparent to those having ordinary 
skill in the art upon examination of the following or may be learned from practice of the 
invention. The objects and advantages of the invention may be realized and attained as 
particularly pointed out in the appended claims. 

To achieve the objects and in accordance with the purposes of the invention, as 
embodied and broadly described herein, a video browsing system for browsing a video based 
on a data structure in which a constant and variable relations between characters of a video 
are connected with characters and corresponding event segments, comprises (a) browsing 
and displaying a video segment showing a constant relation between characters and variable 
relations between characters based on an event segment showing a corresponding constant 
relation and variable relation, and (b) displaying a video segment of a corresponding event 
segment in response to a user's selection. 

The video browser of the present invention is characterized in that event segments 
showing a constant relation between characters and changes in the relations is summarized 
and displayed. Also, main events that significandy act on variable relations are connected 
with corresponding variable relations and characters. 

Moreover, the video browser of the present invention is characterized in that constant 
relation and variable relation between the characters are browsed and displayed as a constant 
relation and a -variable relation between characters selected by the user. The constant relation 
and variable relation between characters may also be browsed and displayed as a constant 
relation and a variable relation between every characters related with a selected character. 

In still another video browser of the present invention, the constant relation and 
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variable relation between characters are displayed in a tree structure. The video browser- of 
the present invention may display a constant relation and variable relation between characters 
depending on preset relation types. The relation types may be set in semantic types such as 
'family relation,' 'business relation, 1 and 'social relation'. 

Furthermore, the video browser of the present invention is characterized in that event 
segments showing constant relation and variable relation between characters is displayed as a 
key frame. Finally, the key frame may shew sshow a corresponding relation between 
characters. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention will be described in detail with reference to the following 
drawings in which like reference numerals refer to like elements wherein: 

Fig. 1 shows an example of a video data structure based on character relation 
according to the present invention; 

Fig. 2 shows an example of a video data structure in which events showing character 
relations are summarized in accordance to the present invention; 

Fig. 3 shows a video browser based on the video data structure of Fig. 2; 

Fig. 4 shows a video data structure of an event segment showing variable relations 
between characters; 

Fig. 5 shows a video browser based on the data structure of Fig. 4 in accordance with 
the present invention; 

Fig. 6 shows an example of a video browser according to the present invention; and 
Fig. 7 shows another example of a video browser according to the present invention. 
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DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

Reference will now be made in detail to the preferred embodiments of the present 
invention, examples of which are illustrated in the accompanying drawings. 

Fig. 1 shows a video data structure based on character relations in accordance with the 
present invention for a video browser bas_ed on content. A video browser based on content is 
disclosed in co-pending application 09/239,530, entided "Motional Video Browsing Data 
Structure and Browsing Method Therefor," and is fully incorporated herein. 

Referring to Fig. 1 , a visual description scheme (DS) 101 is divided into a visuali2ation 
DS 102, a syntactic structure DS 103, and a semantic structure DS 104. The visualization DS 
102 is organized into a highlight view DS 105 for displaying a summarized data as a video 
highlight, and a key frame view DS 1 06 for displaying summarized data based on a key frame. 
Because, a video plot can be summarized briefly or with greater amounts of detail, the 
highlight view DS 1 05 enables a display of multi-levels of summarized data with a level 1 07, a 
view type 108 and a time DS 109. Similarly, the key frame view DS 106 enables a display of 
multi-levels of summarized data with a level 110 and a time DS 111. 

Namely, the highlight view DS 105 is organized into a level 107 which l07 T which has 
multiple levels of highlight data based upon a degree of detail in summarizing a video. In 
such case, summary data in each level may include a corresponding period in the time DS 
109 for use in highlight. The key frame view DS 106 is also organized into a level 1 10 which 
has multiple levels of summarized data based upon a degree of detail. In such case, summary 
data in each level may include a corresponding period in the time DS 111 for use as a key 
frame. 
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The syntactic structure DS 103 is used for displaying the actual video and includes 
actual video segments to be displayed. The syntactic structure DS 103 is organized into actual 
video segments in segment DSs 112, and corresponding temporal positions of video 
segments in the time DS 113. 

The semantic structure DS 104 includes additional information describing a video, 
and is organized into an event DS 114 including event information, an object DS 117 
including object information, and an event/ object relation graph DS 119 including 
information describing relations between objects and places, and corresponding events to the 
relations. Namely, the event DS 114 describes events, and the object DS 117 describes 
objects such as characters and places. The event/ object relation graph DS 119 describes a 
constant relation or changes in relation between characters, a relation between object and 
place, or a relation between object and event. 

Here, a constant relation means either a relation between characters that cannot 
change throughout a video, such as a paten tparent to child relation, or a relation which is 
most representative of the relations between characters. 

When an event is selected to display a video segment corresponding to the event, the 
event DS 114 of the semantic structure DS 104 is used: The event DS 114 is divided into a 
Reference to Segment 115 including information necessary for displaying a segment of a 
video corresponding to the selected event and an annotation DS 116 including information 
which connects events with actual positions of the events in a video and information for 
explaining events in a video. Particularly, the event DS- 114 is used in displaying a selected 
event and connects the selected event with a position of the event in a video data through the 
Reference to Segment 115, and the contents of a selected event is annotated through the 
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annotation DS 116. 

The object DS 117 is organized into an object annotation DS 118 including 
information for describing objects such as characters or places. That is, the object DS 1 1 7 is 
used in displaying characters, and detailed information of the characters can be summarized 
or annotated through the annotation DS 118. 

The event/object relation graph DS 119 includes information on a constant relation 
and variable relation between characters. The event/object relation graph DS 119 is 
organized into an entity relation 120 with a return which allows a display of character 
relations in a tree structure in which a relation may have subordinate relations. According to 
the present invention, constant relations between characters are placed in an upper level of 
the tree while changes of relations between the characters are placed in a lower level of the 
tree. 

The entity relation 212 is also divided into a relation 121, a Reference to Object 124, 
and a Reference to Event 125. The relation 121 is organized into a name 122 including 
information on the titles of relations, and a type 123 including information on the nature of 
relations. For example, a nature of relation may be 'family' and a title of relation may be 
'spouse. 1 The Reference to Object 124 connects related characters with each other and the 
Reference to Event 125 connects events which shows particular relations. 

In the above video data structure, the notation above each data such as {0,1 } , {0,* } , 
or {1,*} indicates the number of data for the corresponding data. For example, the notation 
of {0,1 } for the visualization DS 102 indicates that the visual DS 101 can have zero or one 
visualization DS. On the other hand, the notation of {0,*} for the segment DS 1 12 indicates 
that the syntactic structure DS 103 may have from zero to any number of segment DS. 
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Fig. 2 shows a method of using a data structure of Fig. 1 to easily understand and 
browse a video based on a relations between characters. Referring to Fig. 2, a representative 
relation (constant relation) between 'character 1' and 'character 2' is placed at the top of the 
tree structure, and variable relations (relation 1 ~ relation 4) between 'character V and 
'character 2' are placed at the bottom of the tree structure. Also, main events which are 
significant in a change of relation in a variable relation are placed between variable relations. 
The main events are summarized video data and are displayed by highlight data. 

Here, the events are connected through the Reference to Event 125 in the entity 
relation 1 20 and the video.segments are connected through the Reference to Segment 1 15 in 
the event DS 114. Thereafter, an actual video segment is displayed by connecting the time 
DS 1 1 3 of the segment DS 1 1 2 with the time DS 1 09 of the level 1 07 in the highlight DS 1 05 
of the visual DS 102. 

Fig. 3 shows an example screen of a video browser based on the video data structure 
of Fig. 2. Referring to Fig. 3, 'character n' of a video is displayed on a character screen 301, 
and a video is displayed on a main screen 302 according to a selection received through a 
user interface 303. 

For example, if a user selects 'character 1' and 'character 3' from the character screen 
301, main events which are significant in bringing about a change in the variable relations 
between 'character 1' and 'character 3 f can be browsed based on the data structures of Figs. 1 
and 2. As a result, contents corresponding to the main events are summarized and displayed 
on the main screen 302. 

Fig. 4 shows another method of using a video data structure including information on 
variable relations of characters and main events significant in the variable relations. 
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Referring to Fig. 4, an-ajrepresentative relation (constant relation) between 'character 
1' and 'character 2' is placed at a top of a tree structure, and variable relations (relation 1 ~ 
relation 4) between 'character l 1 and 'character 2 ! are placed at the bottom of the tree 
structure. Main events (event 1 ~ event 3) which are significant in bringing about a change in 
the variable relations are placed between the variable relations. The method of Fig. 4 is 
realized in such a manner that a main event segment which brings a new relation is connected 
with the variable relations through the Reference to Event 125 of the entity relation 120. 

Fig. 5 shows a video browser in accordance with the present invention based on the 
data structure of Fig. 4. Referring to Fig. 5, characters in a video are displayed on a character 
screen 501, and connection between variable relations and events are displayed on a character 
relation-variable event screen 502. Also, an event segment corresponding to a variable 
relation can be reproduced and displayed on a main screen 503. 

For example, if a viewer selects 'character 1' and 'character 3/ variable relations 
between 'character 1' and 'character 3, f and main events which are significant in bringing 
about a change in the variable relation are respectively displayed on the screen 502. At this 
time, a relation or event may be displayed by an-ajtey frame or annotation. Also, a video 
segment corresponding to an event such as 'event 2 ? selected from the screen 502 is 
reproduced and displayed on the main screen 503. 

Fig. 6 shows another example screen of a video browser according to the present 
invention. Referring to Fig. 6, main characters of a video are displayed on a character screen 
601 . Also, characters having relations with a character selected from the character screen 601, 
and a constant relation and variable relations between the selected character and related 
characters are displayed on a relation screen 602. 
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At this time, a constant relation between the selected character and a related character 
is displayed on the top of a relation tree structure while variable relations between the 
selected character and the related character is displayed on the bottom of the tree structure. 

Furthermore, a key frame of events significant in both the constant relation and 
variable relations is displayed on a main scene screen 603. Here, key frames of event segment 
corresponding to a type(s) of event may be displayed on the main scene screen according to a 
user selection through a selection screen 604. Namely, key frames of main event segments 
whieh segments T which directly show a selected relation or key frames of secondary event 
segments which segments. which indirecdy show the selected relation, or both can be 
displayed according to the user selection. For example, if a selected relation is a husband and 
wife, a main event segment may be a video segment of a marriage ceremony while a 
secondary event segment may be a video segment of third parties discussing the marriage 
ceremony. 

An event segment corresponding to an event, for example a key frame, selected from 
the main scene screen 603 can thus be reproduced and displayed on the main screen 605. 

For example, when a user selects 'character l f from among the characters in the 
character screen 601, other characters 'character 2 1 ~ 'character 4' related with 'character V 
are displayed on the relation screen 602. If 'relation 2' with 'character 2 f is selected from the 
relation screen 602, event segments corresponding to 'relation 2' with 'character 2' is 
displayed on the main scene screen 603 as key frames. Also, a period of a video 
corresponding to 'event 6 f selected from the main scene screen 603 is reproduced and 
displayed on the main screen 605. 

At this time, the video browser of Fig. 6 can be implemented by setting a relation type 
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in the data structure of Fig. 1. In other words, events connected with a relation may vary 
depending on whether the type 123 of the relation 121 is the direct or indirect type as 
discussed above. Also, the events are connected through the Reference to Event 125 of the 
entity relation 120 and event segments are connected through the Reference to Segment 115 
of the event DS 114. Thereafter, an actual video segment is displayed by connecting the time 
DS 1 13 of the segment DS 1 12 with the time DS 1 1 1 of the level 1 10 in the key frame view 
DS 1 06 of the visual DS 1 02. 

Fig. 7 shows still another example screen of a video browser according to the present 
invention. Referring to Fig. 7, characters of a video are displayed on a character screen 701; 
and character having relations with a selected character, and a constant relation and a variable 
relation between the selected character and related characters are displayed on a relation 
screen 702. Here, the constant and variable relations may categorized into different natures 
of relations and relations corresponding to a particular nature(s) of relation selected through a 
selection screen 703 may be displayed in the relation screen. The nature of relation may be a 
family relation, a business relation, or a social relation. 

Thus, a key frame of events significant in the selected nature(s) of relation is displayed 
on a main scene screen 704. A video segment corresponding to the selected relation and 
event is reproduced and displayed on a main screen 705. For example, when a user selects 
'character V from characters displayed on the character screen 701, characters (character 2' ~ 
'character 4') having a social relation with 'character V may be displayed and an event 
segment of 'event 6' corresponding to 'social relation 2 f with 'character 2' can be reproduced 
and displayed on the main screen 705. 

As in Fig. 6, a constant relation between a selected character and a related character is 
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displayed on the top of a relation tree structure while variable relations between the selected 
character and the related character is displayed on the bottom of the tree structure. 

The video browser of Fig. 7 can be realized by setting a relation type 123 of the 
relation data 121 in the data structure of Fig. 1. In other words, events related with the 
relation data 121 may vary depending on whether the relation type 123 of the relation data 
121 is a family relation, 1 a 'business relation, 1 or a 'social relation.' As a result, browsing of 
Fig. 7 can be performed. At this time, the events are mutually connected through the 
reference to event 125 of the real relation 120. Segments are mutually connected through the 
reference to segment 115 of the event DS 114. Then, the time DS 113 of the segment DS 
1 12 is connected with the time DS 1 1 1 of the level 107 displayed in the key frame DS 106 of 
the visual DS 102. 

As aforementioned, the video browser based on a character relation according to the 
present invention has the following advantages. The event periods of the variable relation 
between the characters are summarized and displayed, and the main events concerned in the 
variable relation between the characters are connected between the variable relations and 
then displayed. 

Furthermore, the data structure for browsing a video is based on a relation between 
characters and a variable relation between them. A key frame of an event period showing a 
corresponding relation between the characters clearly or directly, or a key frame of an event 
period showing it with hint or indirecdy can be displayed based on the data structure. Also, 
the relations between the characters are divided into a family relation, a social relation, and an 
emotion relation to be selectively provided to a user, so that video browsing can be 
performed based on the relations between the characters. 
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Accordingly, contents of the video can be browsed based on characters that 
significantly act on development of a story of a movie or drama, the relation between the 
characters, and the variable relation between them. After all, the user can easily browse the 
video based on characters, event and relation which are concerned in development of a story 
and important for contents of a real video. Also, the user can understand and browse a 
desired video in various types based on the video data structure which represents characters 
and their relation. 

Furthermore, the present invention may be applied to a VOD system in the 
broadcasting field so that the user can view a desired part. This could lead to an advantage 
that the user can reproduce and view the desired part within short time and to effective 
utilization of the network source. Also, the present invention may be applied to a video 
player for home use and a broadcasting video player so that the video browsing environment 
can be provided to easily browse a desired part of a movie or drama stored in the video 
player. 

The foregoing embodiments are merely exemplary and are not to be construed as 
limiting the present invention. The present teachings can be readily applied to other types of 
apparatuses. The description of the present invention is intended to be illustrative, and not to 
limit the scope of the claims. Many alternatives, modifications, and variations will be 
apparent to those skilled in the art. 
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